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Abstract Body. 


Background / Context. Description of prior research and its intellectual context. 

Recent efforts to attract and retain effective educators and to improve teaching practices 
have focused on reforming evaluation and compensation systems for teachers and principals. In 
2006, Congress established the Teacher Incentive Fund (TIF), which provides grants to support 
performance-based compensation systems for teachers and principals in high-need schools. The 
incentives and support offered through TIF grants aim to improve student achievement by 
improving educator effectiveness and the quality of the teacher workforce. 

Few studies of U.S. pay-for-performance programs have found consistent impacts on 
student achievement. Most experimental studies found no statistically significant impact of pay- 
for-performance on student achievement. This includes studies of New York City’s School-Wide 
Performance Bonus Program (Marsh et al. 2011; Fryer 2011; Goodman and Turner 2010); a 
study of a comprehensive teacher pay reform model in Chicago (Glazerman et al. 2009; 
Glazerman and Seifullah 2010, 2012); and a study in Nashville that offered substantial pay-for- 
performance bonuses ($5,000 to $15,000) to middle school math teachers (Springer et al. 2010). 

However, these studies included schools from only one school district, making it difficult 
for policymakers to determine whether the study findings can be generalized more broadly. In 
addition, the programs in these studies tended to have one or more design weaknesses, for 
example, the average and maximum pay-for-performance bonuses may have been too small to 
provide meaningful incentives for teachers to change their practices (Glazerman et al. 2009; 
Glazerman and Seifullah 2010, 2012). 

Purpose / Objective / Research Question / Focus of Study. Description of the focus of the 
research. 

The number of TIF grants funded by the U.S. Department of Education grew significantly in 
2010, with 62 grants awarded that year compared to 16 grants in 2006 and 18 in 2007. As part of 
the additional funding used to support these grants, Congress required a rigorous evaluation of 
the 2010 TIF grantees. The purpose of the multiyear study is to describe the program 
characteristics and implementation experiences of all 2010 TIF grantees and estimate the impact 
of pay-for-performance bonuses within a well-implemented performance-based compensation 
system for evaluation grantees. Because educators’ understanding of and responses to this policy 
can change over time, this study plans to follow the grantees for the duration of the five-year 
grants. 

This report addresses two research questions: 

1. What are the characteristics of all TIF grantee districts and their performance-based 
compensation systems? What implementation experiences and challenges did TIF 
districts encounter? 

2. How do teachers and principals in schools that did or did not offer pay-for-performance 
bonuses compare on key dimensions, including their understanding of TIF program 
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features, exposure to TIF-funded activities, allocation of time, and attitudes toward 
teaching and the TIF program? 

Setting. Description of the research location. 

The 2010 TIF grant application notice differed from the other rounds of the TIF grants in an 
important way: it included a main competition and an evaluation competition. Applicants had to 
apply for one or the other. By holding two separate competitions, ED created a sample of 
grantees that, by virtue of having applied for an evaluation grant, had indicated their interest and 
willingness to participate in an evaluation to measure the impact of pay-for-performance bonuses 
on educators’ and students’ outcomes. 

Population / Participants / Subjects. Description of the participants in the study: who, how many, 
key features, or characteristics. 

The final study sample for this report consisted of 153 TIF 2010 grantee districts, composed 
of 141 non-evaluation districts and 12 evaluation districts. For 10 evaluation districts, we also 
provide information about the experiences, behaviors, and attitudes of educators. The evaluation 
districts include 137 study schools in which all principals and a sample of 826 teachers were 
administered surveys. 

Compared with all U.S. districts, TIF districts were significantly larger, were more likely to 
be located in urban areas, had a higher proportion of disadvantaged and minority students, were 
more heavily located in the South, and were less likely to be in states with collective bargaining 
requirements. 

Intervention / Program / Practice. Description of the intervention, program, or practice, including 
details of administration and duration. 

The 2010 TIF grants provided more detailed guidance than prior rounds on the measures 
used to evaluate educators and on the design of the pay-for-performance bonuses. The 2010 
grants required four program components in their performance-based compensation systems: 1) 
measures of educator effectiveness that included students’ achievement growth and observations 
of classroom or school practices; 2) pay-for-performance bonus designed to incentivize and 
reward educators solely for being effective; 3) additional pay opportunities for educators to take 
on additional roles or responsibilities such as becoming a master or mentor teacher; and 4) 
professional development to inform teachers about the performance measures and to provide 
support based on educators’ performance on the measures. 

This study focuses most heavily on one of those requirements: the impact of pay-for- 
performance bonuses. To provide a strong incentive to the most effective educators, bonuses 
were to be differentiated and substantial enough to lead to a change in the behavior of teachers 
and principals to improve student outcomes. 


Research Design. Description of the research design. 
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This study examined TIF implementation for all 2010 grantees as well as implementation and 
impacts for a subset of 12 districts selected through the evaluation competition. For evaluation 
districts, the study measured the impact of pay-for-performance bonuses as part of a 
comprehensive reform system within a large, multisite random assignment study design. In each 
evaluation district, schools serving grades 4 through 8 were sorted by lottery into two groups. 
The treatment schools were to implement all components of their performance -based 
compensation system. The control schools implemented the same performance-based 
compensation system with one exception — the pay-for-performance bonus was replaced with an 
across-the-board one percent bonus for all educators (see Figure 1). Because the two groups of 
schools were assigned by lottery within each district, the only difference between the two groups 
was that educators in treatment schools were eligible to earn a pay-for-performance bonus, while 
educators in control schools received an automatic bonus regardless of their performance. All 
other aspects of the program and the school characteristics (measured and unmeasured) were 
similar by design. 

Data Collection and Analysis. Description of the methods for collecting and analyzing data. 

Data on TIF implementation in all districts. To describe TIF program features and 
implementation experiences of TIF districts in general, the study team administered a survey to 
all 2010 TIF district administrators in December 2011. 

Additional data on TIF implementation in 10 evaluation districts. The study team 
supplemented data obtained from the district surveys with information from additional sources to 
describe in more detail TIF programs and implementation experiences in evaluation districts. We 
conducted telephone interviews with staff in evaluation districts in summer 2012. Technical 
assistance documents included needs assessments conducted in fall 2010 and spring 2011, and 
communication materials used by districts and grantees during the 2010-201 1 planning year. 

Data on teachers ’ and principals’ attitudes and behaviors in 10 evaluation districts. The study 
team used teachers’ and principals’ survey responses to examine their understanding of the TIF 
program in their districts and to estimate the impact of pay-for-performance bonuses on their 
attitudes and behaviors. These surveys were administered to all principals in the evaluation 
schools and a sample of teachers in treatment and control schools in spring 2012. The teacher 
sample included all 1st- and 4th-grade teachers, and 7th-grade math, English language arts, and 
science teachers. 

Findings / Results. Description of the main findings with specific details. 

This report describes programs implemented during the 201 1-2012 school year by the 2010 TIF 
grantees. The main findings for all TIF districts include the following: 

• Fewer than half of districts reported implementing all required components of the 
TIF program, evidence that full implementation is a challenge. Although 85 percent 
of TIF districts reported implementing at least three of the four required components for 
teachers, slightly fewer than half (46 percent) reported implementing all four. 
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• Consistent with the TIF grant goals, grantees expected pay-for-performance 
bonuses to be somewhat substantial and differentiated. However, districts expected 
most educators would receive a bonus, suggesting that the award criteria were not 
consistent with TIF guidance for challenging pay-for-performance bonuses. TIF 

districts expected to award an average pay-for-performance bonus of about 4 percent of 
the average U.S. educators’ salary (Figure 2). The maximum bonus expected by TIF 
districts was twice as large as the average bonus for teachers and 50 percent larger than 
the average bonus for principals. Districts also expected to award a pay-for-performance 
bonus to more than 90 percent of eligible teachers and principals (Figure 3). 

The key findings for the 10 evaluation districts that conducted the lottery by spring 2012 include 
the following: 

• Many educators misunderstood the performance measures and the pay-for- 
performance bonuses used for TIF. For example, the measures that educators indicated 
were used to evaluate their performance sometimes differed from those reported by 
districts (Table 1). In addition, more than half of teachers did not know they were eligible 
for pay-for-performance bonuses, and teachers reported a maximum pay-for-performance 
bonus that was lower than the amount reported by districts (Figure 4). 

• Most teachers and principals are satisfied with their professional opportunities, 
school environment, and the TIF program. About two-thirds of teachers were satisfied 
with their jobs overall and were glad to be participating in the TIF program (Table 2). 

• Educators in schools that offered pay-for-performance bonuses tended to be less 
satisfied than those in schools that did not offer such bonuses. For example, fewer 
teachers in schools that offered bonuses were satisfied with the opportunities for 
professional advancement (68 versus 76 percent) and school morale (48 versus 55 
percent) (Table 2). However, more teachers in schools offering pay-for-performance 
bonuses were satisfied with the opportunity to earn additional pay (64 versus 59 percent). 

Conclusions. Description of conclusions, recommendations, and limitations based on findings. 

This study was designed to provide implementation information for the 2010 TIF grantees. 
For the subset of grantees that received evaluation grants, the evaluation examines the impact of 
pay-for-performance bonuses as part of a comprehensive reform system within a large, multisite 
random assignment study design. Targeted technical assistance supported program 
implementation in the 12 evaluation districts to help ensure the proper implementation of their 
performance-based compensation systems. 

Future reports will describe the impact of pay-for-performance bonuses on student 
achievement and teacher mobility, and examine the features of performance-based compensation 
systems that are related to these outcomes. 
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Appendix B. Tables and Figures 
Figure 1. Random Assignment Evaluation Design 
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Figure 2. Average, Minimum, and Maximum Expected Pay-for-Performance Bonuses for Teachers 
and Principals 



Source: District survey. 


Notes: The figure is based on answers to a question about the expected distribution of pay-for-performance 

bonuses, given 10 categories of bonus amounts that range from $0 to $15,000 or more (for example, 
the percentage of teachers expected to earn a bonus between $1 ,000 and $1 ,999). Eighty-seven of the 
153 TIF districts responded to the question for teachers; 99 of the 153 districts responded to the 
question for principals. The maximum bonus by district was calculated as the top range of the largest 
category with a positive percentage of teachers or principals expected to receive a bonus in that range. 
The minimum bonus by district was calculated as the bottom range of the lowest category with a 
positive percentage of teachers or principals expected to receive a bonus in that range. The average 
bonus by district was calculated as the average of the midpoint dollar amount of each category, 
weighted by the percentage of teachers or principals expected to received a bonus in that range. 
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Figure 3. Expected Distribution of Teacher Pay-for-Performance Bonuses in Tested Grades and Subjects 
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Amount of Pay-for-Performance Bonus 


Source: District survey. 

Note: 87 districts responded to this survey question. 


Table 1. Performance Measures Used to Evaluate Teachers and Principals, as Reported by Educators and 
District Representatives 



Percentage of Respondents Reporting the Measure 
Was Used 


Teacher Report 

Principal Report 

District Report 

Teacher Performance Measures 
Student achievement growth 
Classroom observations 

68.0*+ 

78.1*+ 

56.3* 

97.5 

100.0 

100.0 

Sample Size— Range 3 

809-811 

133-134 

10 

Principal Performance Measure 
Student achievement growth for the school 

n.a. 

88.7* 

100.0 

Sample Size 

n.a. 

127 

10 

Sources: Teacher, principal, and district surveys. 

a Sample sizes are presented as a range based on the data available for each row in the table. 

*Difference from the district report is statistically significant at the 0.05 level, two-tailed test. 

+Difference between teacher and principal reports is statistically significant at the 0.05 level, two-tailed test. 


n.a. = not applicable. 
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Figure 4. Maximum Possible Size of Pay-for-Performance Bonuses for Treatment Teachers and 
Principals, as Reported by Respective Educators and Districts 



■ Educator 
Report 

District 

Report 


Sources: Teacher, principal, and district surveys. 


Note: Figures indicate respondents’ average report of the maximum possible size of teachers’ or 

principals’ pay-for-performance bonuses. A total of 395 treatment teachers and 67 treatment 
principals responded to this survey question from 10 of the evaluation districts. 


Table 2. Teachers’ Satisfaction with Performance Measures, Professional Opportunities, School 
Environment, and TIF Program 



Treatment 

Control 

Impact 

Attitudes Toward Aspects of Teaching 

Percentage Who Are Somewhat 
or Very Satisfied 


Classroom Observations as an Evaluation Measure 

68.4 

77.0 

-8.6* 

Opportunities for Professional Advancement 

67.8 

75.7 

-7.8* 

Quality of Interaction with Colleagues 

73.6 

80.6 

-7.0* 

School Morale 

48.1 

54.9 

-6.8* 

Opportunities to Earn Extra Pay 

64.0 

58.9 

5.1* 

Number of Teachers — Range 3 

405-408 

405-412 


Attitudes Toward TIF Program 

Percentage Who Agree or 
Strongly Agree 


My Job Satisfaction Has Increased Due to the TIF Program 

27.1 

32.0 

-4.9* 

The TIF Program Is Fair 

53.0 

57.6 

-4.6* 

1 Feel Increased Pressure to Perform Due to the TIF 
Program 

62.9 

54.1 

8.7* 

Number of Teachers — Range 3 

399-403 

394-403 



Source: Teacher survey. 


a Sample sizes are presented as a range based on the data available for each row in the table. 
*lmpact is statistically significant at the .05 level, two-tailed test. 
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